Unmanned vehicle recognition and threat management

ABSTRACT

Systems and methods for automated unmanned aerial vehicle recognition. A multiplicity of receivers captures RF data and transmits the RF data to at least one node device. The at least one node device comprises a signal processing engine, a detection engine, a classification engine, and a direction finding engine. The at least one node device is configured with an artificial intelligence algorithm. The detection engine and classification engine are trained to detect and classify signals from unmanned vehicles and their controllers based on processed data from the signal processing engine. The direction finding engine is operable to provide lines of bearing for detected unmanned vehicles.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application relates to and claims priority from the followingapplications. This application is a continuation of U.S. patentapplication Ser. No. 17/190,048 filed Mar. 2, 2021, which is acontinuation of U.S. patent application Ser. No. 16/732,811 filed Jan.2, 2020, which is a continuation of U.S. patent application Ser. No.16/275,575 filed Feb. 14, 2019, which claims the benefit of U.S.Provisional Application 62/632,276 filed Feb. 19, 2018. U.S. patentapplication Ser. No. 16/275,575 also claims priority from and is acontinuation-in-part of U.S. patent application Ser. No. 16/274,933filed Feb. 13, 2019, which is a continuation-in-part of U.S. patentapplication Ser. No. 16/180,690 filed Nov. 5, 2018, which is acontinuation-in-part of U.S. patent application Ser. No. 15/412,982filed Jan. 23, 2017. U.S. patent application Ser. No. 16/180,690 alsoclaims priority from U.S. Provisional Patent Application No. 62/722,420filed Aug. 24, 2018. U.S. patent application Ser. No. 16/274,933 alsoclaims the benefit of U.S. Provisional Application 62/632,276 filed Feb.19, 2018. Each of the above-mentioned applications is incorporatedherein by reference in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to systems and methods for unmannedvehicle recognition and threat management. More particularly, thesystems and methods of the present invention are directed to unmannedvehicle detection, classification and direction finding.

2. Description of the Prior Art

Unmanned Aerial Vehicles (UAVs), commonly known as drones, have becomereadily available in commercial and retail stores. Detailed schematicsfor their control systems are available from many manufacturers and theinternet along with Software Development Kits (SDKs). Rapidmodifications are enabled by advancements in various technologies (e.g.,3D printing). UAVs can be modified to deploy dangerous actions andthreaten societal securities. For example, UAVs can be modified todeliver dangerous payloads. It is no longer a question of if, it is nowa question of when. Thus, it is imperative that organizations andgovernments take steps to protect critical assets (e.g., ports, powerplants), structures (e.g., buildings, stadiums), and personnel and theircitizens.

Exemplary U.S. patent Documents relevant to the prior art include:

U.S. Pat. No. 9,862,489 for “Method and apparatus for drone detectionand disablement” by inventors Lee Weinstein et al., filed Feb. 7, 2016and issued Jan. 9, 2018, describes a method and apparatus for detectionand disablement of an unidentified aerial vehicle (UAV) includes arraysof antenna elements receiving in two modalities (for instance radiofrequency (RF) and acoustic modalities, or RF and optical modalities).Signal processing of outputs from multiple antenna arrays locates apotential UAV at specific coordinates within a volume of space undersurveillance, and automatically aims video surveillance and ashort-range projectile launcher at the UAV, and may automatically firethe projectile launcher to down the UAV.

U.S. Pat. No. 9,858,947 for “Drone detection and classification methodsand apparatus” by inventors Brian Hearing et al., filed Nov. 24, 2015and issued Jan. 2, 2018, describes a system, method, and apparatus fordrone detection and classification. An example method includes receivinga sound signal in a microphone and recording, via a sound card, adigital sound sample of the sound signal, the digital sound samplehaving a predetermined duration. The method also includes processing,via a processor, the digital sound sample into a feature frequencyspectrum. The method further includes applying, via the processor, broadspectrum matching to compare the feature frequency spectrum to at leastone drone sound signature stored in a database, the at least one dronesound signature corresponding to a flight characteristic of a dronemodel. The method moreover includes, conditioned on matching the featurefrequency spectrum to one of the drone sound signatures, transmitting,via the processor, an alert.

U.S. Pat. No. 9,767,699 for “System for and method of detecting drones”by inventors John W. Borghese et al., filed May 14, 2015 and issued Sep.19, 2017, describes an apparatus and method can provide a warning of adrone or unmanned aerial vehicle in the vicinity of an airport. Theapparatus can include at least one antenna directionally disposed at analong the approach or departure path and a detector configured toprovide a warning of a presence of sense an unmanned aerial or drone.The warning can be provided in response to a radio frequency signalreceived by the at least one of the antenna being in a frequency bandassociated with a transmission frequency for the unmanned aerial vehicleor drone or in a frequency band associated with interaction from receivecircuitry of the unmanned aerial vehicle or drone.

U.S. Pat. No. 9,715,009 for “Deterent for unmanned aerial systems” byinventors Dwaine A. Parker et al., filed Dec. 2, 2016 and issued Jul.25, 2017, describes a system for providing an integrated multi-sensordetection and countermeasure against commercial unmanned aerialsystems/vehicles and includes a detecting element, a tracking element,an identification element, and an interdiction element. The detectingelement detects an unmanned aerial vehicle in flight in the region of,or approaching, a property, place, event or very important person. Thetracking element determines the exact location of the unmanned aerialvehicle. The identification/classification element utilizing data fromthe other elements generates the identification and threat assessment ofthe UAS. The interdiction element, based on automated algorithms caneither direct the unmanned aerial vehicle away from the property, place,event or very important person in a non-destructive manner, or candisable the unmanned aerial vehicle in a destructive manner. Theinterdiction process may be over ridden by intervention by a SystemOperator/HiL.

U.S. Pat. No. 9,529,360 for “System and method for detecting anddefeating a drone” by inventors Howard Melamed et al., filed Apr. 22,2015 and issued Dec. 27, 2016, describes a system for detecting anddefeating a drone. The system has a detection antenna array structuredand configured to detect the drone and the drone control signal over a360 degree field relative to the detection antenna array includingdetecting the directionality of the drone. The system also includes aneutralization system structured and configured in a communicatingrelation with the detection antenna array. The neutralization system hasa transmission antenna structured to transmit an override signal aimedat the direction of the drone, an amplifier configured to boost the gainof the override signal to exceed the signal strength of the dronecontrol signal, and a processing device configured to create and effectthe transmission of the override signal. The patent also discloses amethod for detecting and defeating a drone.

U.S. Publication No. 2017/0358103 for “Systems and Methods for TrackingMoving Objects” by inventors Michael Shao et al., filed Jun. 9, 2017 andpublished Dec. 14, 2017, describes systems and methods for trackingmoving objects. The publication discloses an object tracking systemcomprises a processor, a communications interface, and a memoryconfigured to store an object tracking application. The object trackingapplication configures the processor to receive a sequence of images;estimate and subtract background pixel values from pixels in a sequenceof images; compute sets of summed intensity values for different perframe pixel offsets from a sequence of images; identify summed intensityvalues from a set of summed intensity values exceeding a threshold;cluster identified summed intensity values exceeding the thresholdcorresponding to single moving objects; and identify a location of atleast one moving object in an image based on at least one summedintensity value cluster.

U.S. Publication No. 2017/0261613 for “Counter drone system” by inventorBrian R. Van Voorst, filed Feb. 27, 2017 and published Sep. 14, 2017,describes a counter drone system that includes a cueing sensor to detectthe presence of an object wherein the cueing sensor cues the presence ofa target drone, a long range LIDAR system having a sensor pointed in adirection of the target drone to acquire and track at long range thetarget drone to provide an accurate location of the target drone whereinonce a track is acquired, the motion of the target drone is used tomaintain the track of the target drone and a threat detector whereinLIDAR data is provided to the threat detector to determine if the targetdrone is a threat.

U.S. Publication No. 2017/0261604 for “Intercept drone tasked tolocation of lidar tracked drone” by inventor Brian Van Voorst, filedFeb. 27, 2017 and published Sep. 14, 2017, describes a system thatincludes a long range LIDAR tracking system to track a target drone andprovide detection and tracking information of the target drone; acontrol system to process the detection and tracking information andprovide guidance information to intercept the target drone; and a highpowered intercept drone controlled by supervised autonomy, thesupervised autonomy provided by processing the detection and trackinginformation of the target drone and sending guidance information to theintercept drone to direct the intercept drone to the target drone.

U.S. Publication No. 2017/0039413 for “Commercial drone detection” byinventor Gary J. Nadler, filed Aug. 3, 2015 and published Feb. 9, 2017,describes a method of capturing the presence of a drone, including:collecting, using at least one sensor, data associated with an aerialobject; analyzing, using a processor, the data to determine at least onecharacteristic of the aerial object; accessing, in a database, a libraryof stored characteristics of commercially available drones; determining,based on the analyzing, if the at least one characteristic of the aerialobject matches a characteristic of a commercially available drone; andresponsive to the determining, generating an indication of a positivematch.

SUMMARY OF THE INVENTION

The present invention provides systems and methods for unmanned vehiclerecognition. In one embodiment, a multiplicity of receivers captures RFdata and transmits the RF data to at least one node device. The at leastone node device comprises a signal processing engine, a detectionengine, a classification engine, and a direction finding engine. The atleast one node device is configured with an artificial intelligencealgorithm. The detection engine and classification engine are trained todetect and classify signals from unmanned vehicles and their controllersbased on processed data from the signal processing engine. The directionfinding engine is operable to provide lines of bearing for detectedunmanned vehicles. A display and control unit is in networkcommunication with the at least one node device for displaying locationsand other related data for the detected unmanned vehicles.

These and other aspects of the present invention will become apparent tothose skilled in the art after a reading of the following description ofthe preferred embodiment when considered with the drawings, as theysupport the claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system for unmanned vehicle recognition according to oneembodiment of the present invention.

FIG. 2 illustrates signal characterization within a spectrum from 700MHz to 900 MHz.

FIG. 3 is an illustration of Phantom 4 controller and drone signals.

FIG. 4 is a human interface display for drone detection according to oneembodiment of the present invention.

FIG. 5 shows a setup for RF data capture in an Anechoic Chamberaccording to one embodiment of the present invention.

FIG. 6 illustrates a simulation for fading and channel hopping of anOcuSync drone.

FIG. 7 is an illustration of an inception based convolutional neuralnetwork.

FIG. 8 illustrates a scenario for an RF environment with just noise.

FIG. 9 illustrates a scenario for an RF environment with a Phantom 4controller and drone.

FIG. 10 illustrates a scenario for an RF environment with two Mavic Prodrones.

FIG. 11 illustrates a scenario for an RF environment with a Mavic Procontroller only.

FIG. 12 illustrates a scenario for an RF environment with one Mavic Prodrone only.

FIG. 13 illustrates a scenario for an RF environment with one Phantom 3controller only.

FIG. 14 illustrates a scenario for an RF environment with a Phantom 3controller and drone.

FIG. 15 illustrates a scenario for an RF environment with a Mavic Procontroller and drone in wideband mode.

FIG. 16 illustrates a scenario for an RF environment with a Mavic Prodrone in wideband mode.

FIG. 17 illustrates a scenario for an RF environment with a Mavic Procontroller and drone.

FIG. 18 illustrates a scenario for an RF environment with a Mavic Procontroller and a Phantom 4 controller.

FIG. 19 is an illustration of identifying drones and controllers basedon signal edge detection.

FIG. 20 is an illustration with averaged signal amplitudes and signaledges according to one embodiment of the present invention.

FIG. 21 displays a detection range of less than 500 meters based onequipment specification and location.

DETAILED DESCRIPTION

The present invention provides systems and methods for unmanned vehiclerecognition. The present invention relates to automatic signaldetection, temporal feature extraction, geolocation, and edge processingdisclosed in U.S. patent application Ser. No. 15/412,982 filed Jan. 23,2017, U.S. patent application Ser. No. 15/478,916 filed Apr. 4, 2017,U.S. patent application Ser. No. 15/681,521 filed Aug. 21, 2017, U.S.patent application Ser. No. 15/681,540 filed Aug. 21, 2017, and U.S.patent application Ser. No. 15/681,558 filed Aug. 21, 2017, each ofwhich is incorporated herein by reference in its entirety.

Currently, commercial and retail UAVs dominate frequencies including 433MHz industrial, scientific, and medical radio band (ISM Band) Region 1,900 MHz ISM Band Region 1,2,3 (varies by country), 2.4 GHz (channels1-14), 5 GHz (channels 7-165 most predominant), and 3.6 GHz (channels131-183). Modulation types used by commercial and retail UAVs includeDirect Sequence Spread Spectrum (DSSS), Orthogonal Frequency DivisionMultiplexing (OFDM), Frequency Hopping Spread Spectrum (FHSS), FatabaAdvanced Spread Spectrum Technology (FASST).

Many counter UAV systems in the prior art focus on the 2.4 GHz and 5.8GHz bands utilizing demodulation and decryption of radio frequency (RF)signals to detect and analyze each signal to determine if it is relatedto a UAV.

The present invention provides systems and methods for unmanned vehiclerecognition including detection, classification and direction finding.Unmanned vehicles comprise aerial, terrestrial or water borne unmannedvehicles. The systems and methods for unmanned vehicle recognition areoperable to counter threats from the aerial, terrestrial or water borneunmanned vehicles.

In one embodiment, a multiplicity of receivers captures RF data andtransmits the RF data to at least one node device. The at least one nodedevice comprises a signal processing engine, a detection engine, aclassification engine, and a direction finding engine. The at least onenode device is configured with an artificial intelligence algorithm. Thedetection engine and classification engine are trained to detect andclassify signals from unmanned vehicles and their controllers based onprocessed data from the signal processing engine. The direction findingengine is operable to provide lines of bearing for detected unmannedvehicles. A display and control unit is in network communication withthe at least one node device for displaying locations and other relateddata for the detected unmanned vehicles.

In one embodiment, the present invention provides systems and methodsfor unmanned vehicle (UV) recognition in a radio frequency (RF)environment. A multiplicity of RF receivers and a displaying device arein network communication with a multiplicity of node devices. Themultiplicity of RF receivers is operable to capture the RF data in theRF environment, convert the RF data to fast Fourier transform (FFT)data, and transmit the FFT data to the multiplicity of node devices. Themultiplicity of node devices each comprises a signal processing engine,a detection engine, a classification engine, a direction-finding engine,and at least one artificial intelligence (AI) engine. The signalprocessing engine is operable to average the FFT data into at least onetile. The detection engine is operable to group the FFT data intodiscrete FFT bins over time, calculate average and standard deviation ofpower for the discrete FFT bins, and identify at least one signalrelated to at least one UV and/or corresponding at least one UVcontroller. The at least one AI engine is operable to generate an outputfor each of the at least one tile to identify at least one UV andcorresponding at least one UV controller with a probability, andcalculate an average probability based on the output from each of the atleast one tile. The classification engine is operable to classify the atleast one UV and/or the at least one UV controller by comparing the atleast one signal to classification data stored in a classificationlibrary in real time or near real time. The direction-finding engine isoperable to calculate a line of bearing for the at least one UV. Thedisplaying device is operable to display a classification of the atleast one UV and/or the at least one UV controller and/or the line ofbearing of the at least one UV. Each of the at least one tile isvisually represented in a waterfall image via a graphical user interfaceon the displaying device.

FIG. 1 illustrates a system for unmanned vehicle recognition accordingto one embodiment of the present invention. The system includes amultiplicity of antennas, a receiver and processing unit, and a displayand control unit. In one embodiment, there are four multibandomnidirectional antennas. In one embodiment, three multibandomnidirectional antennas are positioned to form an equilateral with 6meters spacing as illustrated in FIG. 1 as an example. The receiver andprocessing unit includes a signal processing engine, a UAV detectionengine, a UAV classification engine, a direction finding processingengine, and an internal Global Positioning System (GPS). The receiverand processing unit is operable to receive RF data from the antennas andautomatically process the RF data for UAV detection and classificationand direction finding. The display and control unit includes a humaninterface display. In one embodiment, the human interface display isprovided by a remote web-based interface. The display and control unitis operable to display lines of bearings for detected UAVs andcontrollers, classification for detected UAVs and controllers, receivedsignal strength (RSS) values, and operating frequencies. In oneembodiment, the display and control unit is SigBase 4000 as shown inFIG. 1 . In another embodiment, any computer, laptop, or tabletconfigured with the human interface display of the present invention isoperable to function as a display and control unit. In one embodiment,the receiver and processing unit is a node device, and there aremultiple node devices communicating with each other and forming a groupof networked nodes for UAV detection, classification, and directionfinding.

The present invention provides a more efficient methodology for UAVdetection and identification, which takes advantage of Fast FourierTransform (FFT) over a short period of time and its derivation. RF datareceived from antennas are directly converted to FFT data with finergranularity. This allows rapid identification of protocols used by highthreat drones without demodulation, and the identification isprobability based. An analytics engine is operable to perform nearreal-time analysis and characterize signals within the spectrum underobservation. FIG. 2 illustrates signal characterization within aspectrum from 700 MHz to 900 MHz. Temporal feature extraction is appliedfor signal characterization, which is described in U.S. patentapplication Ser. No. 15/412,982 filed Jan. 23, 2017, U.S. patentapplication Ser. No. 15/681,521 filed Aug. 21, 2017, U.S. patentapplication Ser. No. 15/681,540 filed Aug. 21, 2017, U.S. patentapplication Ser. No. 15/681,558 filed Aug. 21, 2017, each of which isincorporated herein by reference in its entirety.

Advantageously, multiple receivers in the present invention worktogether to ingest spectral activities across large blocks of spectrum.The multiple receivers have an instantaneous bandwidth from 40 MHz to500 MHz. In one embodiment, the multiple receivers are configurable in40 MHz and 125 MHz segment building blocks. Input data are converteddirectly to FFT data and fed into process engines, which significantlydecreases latency. The process engines are designed for rapididentification of signals of interest (SOI). When an SOI is detected, adirection finding process is initiated autonomously. In one embodiment,the direction finding process is configurable by an operator.

There are multiple types of communications links utilized for commandand control of an unmanned vehicle. Although several cost-effectiveradio communication (RC) protocols are gaining global popularity, WI-FIis still the most popular protocol for command and control of UAVs andcamera systems. A remote controller of a UAV acts as a WI-FI accesspoint and the UAV acts as a client. There are several limiting factorsfor WI-FI-based UAVs. For example, the operational range of aWI-FI-based UAV is typically limited to 150 feet (46 m) indoor and 300feet (92 m) outdoor. There is significant latency for control and videobehaviors. Interference by other WI-FI devices affects operationalcontinuity of the WI-FI-based UAVs.

Demand in the UAV user community has made more professional-levelprotocols available in the commercial and retail markets. By way ofexample but not limitation, two common RC protocols used for UAVs areLightbridge and OcuSync. Enhancements in drone technology inevitablyincreases the capability of drones for use in industrial espionage andas weapons for nefarious activities.

Lightbridge is developed for long range and reliable communication.Communication is available within a range up to 5 km. Lightbridgesupports 8 selectable channels, and the selection can be manual orautomatic. Drones with Lightbridge protocol also have the ability toassess interference and move to alternate channels for greater quality.

OcuSync is developed based on the LightBridge protocol. OcuSync useseffective digital compression and other improvements, which decreasesknowledge required to operate. OcuSync provides reliable HD and UHDvideo, and OcuSync-based drones can be operated in areas with greaterdynamic interference. Ocusync improves command and control efficienciesand reduces latency. With OcuSync, video communications are improvedsubstantially, operational range is increased, command and controlrecovery are enhanced when interference occurs.

The systems and methods of the present invention for unmanned vehiclerecognition are operable to detect and classify UAVs at a distance,provide directions of the UAVs, and take defensive measures to mitigaterisks. The detection and classification are fast, which provides moretime to react and respond to threats. Exact detection range is basedupon selection of antenna systems, topology, morphology, and clientcriteria. Classification of the detected UAVs provides knowledge of theUAVs and defines effective actions and capabilities for countering UAVthreats. In one embodiment, the direction information of the UAVsprovides orientation within the environment based on the location of theUAV detector.

In one embodiment, the systems and methods of the present inventionprovides unmanned vehicle recognition solution targeting radiocontrolled and WI-FI-based drones. The overall system is capable ofsurveying the spectrum from 20 MHz to 6 GHz, not just the common 2.4 GHzand 5.8 GHz areas as in the prior art. In one embodiment, the systemsand methods of the present invention are applied to address 2 majorcategories: RC-based UAV systems and WI-FI-based UAV systems. In oneembodiment, UAV systems utilize RC protocols comprising LightBridge andOcuSync. In another embodiment, UAV systems are WI-FI based, for examplebut not for limitation, 3DR Solo and Parrot SkyController. The systemsand methods of the present invention are operable to detect UAVs andtheir controllers by protocol.

The systems and methods of the present invention maintain astate-of-the-art learning system and library for classifying detectedsignals by manufacturer and controller type. The state-of-the-artlearning system and library are updated as new protocols emerge.

In one embodiment, classification by protocol chipset is utilized toprovide valuable intelligence and knowledge for risk mitigation andthreat defense. The valuable intelligence and knowledge includeeffective operational range, supported peripherals (e.g., external orinternal camera, barometers, GPS and dead reckoning capabilities),integrated obstacle avoidance systems, and interference mitigationtechniques.

The state-of-the-art learning system of the present invention is highlyaccurate and capable of assessing detected UAV signals and/or controllersignals for classification in less than a few seconds with a highconfidence level. The state-of-the-art learning system is operable todiscriminate changes in the environment for non-drone signals as well asdrone signals. FIG. 3 is an illustration of Phantom 4 controller anddrone signals. A human interface is operable to display theclassification results. FIG. 4 illustrates a human interface display fordrone detection according to one embodiment of the present invention.

It is difficult to recognize commercial and retail drones with the nakedeye over 100 meters. It is critical to obtain a vector to the target forsituational awareness and defense execution. The systems and methods ofthe present invention provides lines of bearing for direction findingfor multiple UAVs flying simultaneously. Each line of bearing is colorcoded for display. Angles, along with frequencies utilized for uplinkand downlink, are also displayed on the human interface.

Once a UAV is detected and classified, an alert is posted to a counterUAV system operator (e.g., a network operation center, an individualoperator) including azimuth of the UAV and other information. The alertis transmitted via email, short message service (SMS) or third-partysystem integration. The counter UAV system is operable to engage anintercession transmission, which will disrupt the communication betweenthe UAV and its controller. When the communication between the UAV andits controller is intercepted, the UAV will invoke certain safetyprotocols, such as reduce height and hover, land, or return to thelaunch point. The counter UAV system may have certain restrictions basedon country and classification of the UAV.

In one embodiment, the systems and methods of the present invention areoperable to update the UAV library with emerging protocols forclassification purposes, and refine the learning engine for widebandspectrum analysis for other potential UAV signatures, emerging protocolsand technologies. In other words, the systems and methods of the presentinvention are adaptable to any new and emerging protocols andtechnologies developed for unmanned vehicles. In one embodiment,multiple node devices in the present invention are deployed to operateas a group of networked nodes. In one embodiment, the group of networkednodes are operable to estimate geographical locations for unmannedvehicles. In one embodiment, two node devices are operable to provide asingle line of bearing and approximate a geographical location of adetected drone or controller. The more node devices there are in thegroup of network nodes, the more lines of bearing are operable to beprovided, and the more accurate the geographical location is estimatedfor the detected drone or controller. In one embodiment, the geolocationfunction provides altitude and distance of a targeted drone.

In one embodiment, the counter UAV system in the present invention isoperable to alert when unexpected signal characteristics are detected in2.4 GHz and 5.8 GHz areas and classify the unexpected signalcharacteristics as potential UAV activities. In another embodiment, thecounter UAV system in the present invention is operable to alert whenunexpected signal characteristics are detected anywhere from 20 MHz to 6GHz and classify the unexpected signal characteristics as potential UAVactivities. In another embodiment, the counter UAV system in the presentinvention is operable to classify the unexpected signal characteristicsas potential UAV activities when unexpected signal characteristics aredetected anywhere from 40 MHz to 6 GHz. The automatic signal detectionengine and analytics engine are enhanced in the counter UAV system torecognize potential UAV activities across a great portion of thespectrum. In one embodiment, any blocks of spectrum from 40 MHz to 6 GHzare operable to be selected for UAV recognition.

In one embodiment, vector-based information including inclinations,declinations, topology deviations, and user configurable Northing maporientation is added to the WGS84 mapping system for direction findingand location estimation. In one embodiment, earth-centered earth-fixedvector analysis is provided for multi-node systems to estimate UAVlocations, derive UAV velocities from position changes over time, anddetermine UAV trajectory vectors in fixed nodal processing. In oneembodiment, a group of networked node devices are operable tocontinually provide lines of bearing over time, approximate geographicallocations of a detected unmanned vehicle on or above the earth, andtrack the movement of the detected unmanned vehicle from one estimatedlocation to another. In one embodiment, the group of networked nodedevices are operable to determine velocities of the detected unmannedvehicle based on estimated locations and travel time. In one embodiment,the group of networked node devices are operable to estimate atrajectory of the detected unmanned vehicle based on the estimatedgeographical locations over time. In one embodiment, the group ofnetworked node devices are operable to estimate accelerations anddecelerations of the unmanned vehicle based on the velocities of theunmanned vehicles over time.

In one embodiment, the systems and methods of the present invention areoperable for UAV detection and direction finding for differentmodulation schemes including but not limited to DSSS, OFDM, FHSS, FASST,etc. In one embodiment, the counter UAV system in the present inventionis configured with cameras for motion detection. The cameras have bothday and night vision.

In one embodiment, systems and methods of the present invention providestraining for unmanned vehicle recognition. RF data is captured for aPhantom 3 drone and its controller and a Phantom 4 drone and itscontroller, both of which use Lightbridge protocol. RF data is alsocaptured for a Mavic Pro drone and its controller, which uses OcuSyncprotocol. The RF data is recorded at different channels, different RFbandwidths, and different video quality settings inside and outside anAnechoic Chamber. FIG. 5 shows a setup for RF data capture in anAnechoic Chamber according to one embodiment of the present invention.The recordings are overlaid on the RF environment, and fading andchannel hopping are simulated. FIG. 6 illustrates a simulation forfading and channel hopping of an OcuSync drone.

In one embodiment, the recorded RF data is used to train and calibratean inception based convolutional neural network comprised in a dronedetection system. FIG. 7 is an illustration of an inception basedconvolutional neural network. U.S. Patent Publication No. 2018/0137406titled “Efficient Convolutional Neural Networks and Techniques to ReduceAssociated Computational Costs” is incorporated herein by reference inits entirety. The inception based convolutional neural network generatesprobabilities that drones or their controllers are detected. Thedetection probabilities are updated multiple times per second.

The trained inception based convolutional neural network is operable toidentify Lightbridge 1 controller and drone, Lightbridge 2 controllerand drone, and OcuSync controller and drone. The trained inception basedconvolutional neural network is operable to identify Lightbridge andOcusync controllers and drones at the same time. In one embodiment, thedrone detection system comprising the trained inception basedconvolutional neural network is operable to search an instantaneousbandwidth of 147.2 MHz.

In one embodiment, the drone detection system of the present inventionincludes an artificial intelligence (AI) algorithm running on a singleboard computer (e.g., Nvidia Jetson TX2) with an execution time lessthan 10 ms. The drone detection system is operable to separate Phantom 3and Phantom 4 controllers. Waveforms for Phantom 3 and Phantom 4controllers are sufficiently different to assign separate probabilities.

The Artificial Intelligence (AI) algorithm is used to enhanceperformance for RF data analytics. The RF data analytics process basedon the AI algorithm is visualized. The RF waterfalls of several dronescenarios are presented in FIGS. 8-18 . FIG. 8 illustrates a scenariofor an RF environment with just noise. FIG. 9 illustrates a scenario foran RF environment with a Phantom 4 controller and drone. FIG. 10illustrates a scenario for an RF environment with two Mavic Pro drones.FIG. 11 illustrates a scenario for an RF environment with a Mavic Procontroller only. FIG. 12 illustrates a scenario for an RF environmentwith one Mavic Pro drone only. FIG. 13 illustrates a scenario for an RFenvironment with one Phantom 3 controller only. FIG. 14 illustrates ascenario for an RF environment with a Phantom 3 controller and drone.FIG. 15 illustrates a scenario for an RF environment with a Mavic Procontroller and drone in wideband mode. FIG. 16 illustrates a scenariofor an RF environment with a Mavic Pro drone in wideband mode. FIG. 17illustrates a scenario for an RF environment with a Mavic Pro controllerand drone. FIG. 18 illustrates a scenario for an RF environment with aMavic Pro controller and a Phantom 4 controller.

Each scenario is illustrated with 6 waterfall images. Each waterfallrepresents ˜80 ms of time and 125 MHz of bandwidth. The top left imageis the waterfall before an AI processing. The other five images arewaterfalls after the AI processing. For each signal type, the areas ofthe waterfall that are likely for the RF signal type are highlighted.Areas that are not for the signal type are grayed out. The overallprobability that a signal exists in the image is printed in the title ofeach waterfall image. In one embodiment, the AI algorithm is securelyintegrated with a state engine and a detection process of the presentinvention.

In one embodiment, a method for drone detection and classificationcomprises applying FFT function to RF data, converting FFT data intologarithmic scale in magnitude, averaging converted FFT into 256 by 256array representing 125 MHz of bandwidth and 80 ms of time as a basetile, applying normalization function to the base tile, applying aseries of convolutional and pooling layers, applying modified You OnlyLook Once (YOLO) algorithm for detection, grouping bounding boxesdisplayed in the waterfall images (e.g., waterfall plots in FIGS. 8-18), classifying signals based on the shape of detection output, verifyingresults with a second level recurrent neural network (RNN) based patternestimator.

In one embodiment, a method for training comprises recording clean RFsignals, shifting RF signals in frequency randomly, creating truth datafor YOLO output, adding a simulated channel to the RF signals, recordingtypical RF backgrounds, applying FFT function to RF data, converting FFTdata into logarithmic scale in magnitude, averaging converted FFT into256 by 256 array representing 125 MHz of bandwidth and 80 ms of time asa base tile, applying normalization function to the base tile, applyinga series of convolutional and pooling layers, applying modified You OnlyLook Once (YOLO) algorithm for detection, grouping bounding boxesdisplayed in the waterfall images (e.g., waterfall plots in FIGS. 8-18), applying a sigmoid cross entropy function, and applying an AdaptiveMoment Estimation (Adam) based back propagation algorithm.

In one embodiment, a drone detection engine is operable to convert FFTflows from a radio to a tile. For each channel, the drone detectionengine is operable to standardize the FFT output from the radio at adefined resolution bandwidth, and group high resolution FFT data intodistinct bins overtime. The drone detection engine is further operableto calculate average and standard deviation of power for discrete FFTbins, assign a power value to each channel within the tile. Each scan orsingle stare at the radio is a time slice, and multiple time slices withpower and channel assignment create a tile. Tiles from differentfrequency spans and center frequencies are identified as a tile group bya tile group number. Receivers in the drone detection system areoperable to be re-tuned to different frequencies and spans. In oneembodiment, the drone detection system comprises multiple receivers togenerate tiles and tile groups.

In one embodiment, a tile is sent to a YOLO AI Engine. Outputs of adecision tree in the YOLO AI engine are used to detect multiple dronesand their controllers. Drones of the same type of radio protocol areoperable to be identified within the tile. Controllers of the same typeof radio protocol are operable to be identified within the tile. Dronesof different radio protocols are also operable to be identified withinthe tile. Controllers of different radio protocols are also operable tobe identified within the tile. FIG. 19 is an illustration of identifyingdrones and controllers based on signal edge detection.

In one embodiment, a plurality of tiles is sent to the YOLO AI engine.In one embodiment, a tile group is sent to the YOLO AI engine. The YOLOAI engine generates an output for each tile to identify drones and theircontrollers with a probability. An average probability is calculatedbased on outputs for multiple tiles in the tile group. For each tilegroup, the YOLO AI engine computes outputs for several tiles per second.

In one embodiment, a state engine controls the flows of tiles and tilegroups into one or more AI engines. AI engines do not use frequencyvalues for analytics. Thus, the one or more AI engines are operable forany frequency and frequency span that a drone radio supports. The stateengine further correlates output of the one or more AI engines toappropriate tiles and tile groups.

The systems and methods of the present invention are operable fordirection finding of drones and their controllers. Outputs from the AIengine are denoted with time basis for the drones and their controllers.

Drones typically maintain the same frequency unless their firmwaredetects interference. Then the drones may negotiate a change with theircontrollers. This does not create an issue for detection as long as thenew frequency and span is monitored by the systems and methods of thepresent invention. Drone controllers typically use a frequency hoppingspread spectrum (FHSS) or other Frequency hopping system (e.g., Gaussianfrequency shift keying (GFSK)).

In one embodiment, the systems and method of the present invention areoperable to approximate a start time of a line of bearing for adirection finding (DF) system. The time intervals are either known orestimated based upon the behavior monitored by the AI engine and stateengine. This allows the time slice and frequency of each individualdrone and/or controller to be passed to the DF system. In oneembodiment, three or four receivers are coordinated to collectinformation in appropriate frequency segments, wherein the frequencysegments are similar to tiles described earlier. FIG. 20 is anillustration with averaged signal amplitudes and signal edges accordingto one embodiment of the present invention.

The AI engine examines the segments to determine if a drone orcontroller exists. An azimuth of the drone or controller in anEarth-Centered Earth-Fixed coordinate system is determined based onother information collected from the three or four receivers using timedifference of arrival (TDOA), angle of arrival (AOA), power correlative,or interferometry techniques.

Distance capability of UAV detection and classification system dependson hardware configuration, environment morphology and restrictions basedon country and classification of the counter UAV operator. In oneembodiment, the systems and methods for unmanned vehicle recognition areoperable to detect unmanned vehicles within 3-4 kilometers. FIG. 21displays a detection range of less than 500 meters based on equipmentspecification and location.

Certain modifications and improvements will occur to those skilled inthe art upon a reading of the foregoing description. The above-mentionedexamples are provided to serve the purpose of clarifying the aspects ofthe invention and it will be apparent to one skilled in the art thatthey do not serve to limit the scope of the invention. All modificationsand improvements have been deleted herein for the sake of concisenessand readability but are properly within the scope of the presentinvention.

What is claimed is:
 1. A system for unmanned vehicle (UV) recognition ina radio frequency (RF) environment, comprising: at least one node devicein network communication with a multiplicity of RF receivers; whereinthe multiplicity of RF receivers is operable to capture RF data in theRF environment, convert the RF data to fast Fourier transform (FFT)data, and transmit the FFT data to the at least one node device; whereinthe at least one node device comprises a signal processing engine and adetection engine, and wherein the at least one node device is configuredwith an artificial intelligence (AI) algorithm; wherein the signalprocessing engine is operable to average the FFT data into at least onetile; and wherein the detection engine is operable to detect at leastone signal related to at least one UV in the at least one tile based onthe AI algorithm.
 2. The system of claim 1, wherein the AI algorithmcomprises an inception-based convolutional neural network operable togenerate probabilities that UVs are detected.
 3. The system of claim 1,wherein the AI algorithm comprises a You Only Look Once (YOLO) algorithmoperable to receive the at least one tile, generate an output for eachof the at least one tile to identify the at least one UV with aprobability, and calculate an average probability based on the outputfor each of the at least one tile.
 4. The system of claim 1, whereineach of the at least one tile is a 256 by 256 array representing 125 MHzof bandwidth and 80 ms of time.
 5. The system of claim 1, wherein the atleast one node device further comprises a classification engine operableto classify the at least one UV by comparing the at least one signal toclassification data and/or a direction-finding engine operable toestimate a line of bearing of the at least one UV.
 6. The system ofclaim 5, further including a displaying device, wherein the displayingdevice is operable to display a line of bearing of the at least one UVand/or a classification of the at least one UV.
 7. The system of claim1, wherein the RF data is from a spectrum between 20 MHz and 6 GHz. 8.The system of claim 1, wherein the detection engine is operable todetect the at least one UV by radio communication protocols.
 9. Thesystem of claim 1, wherein the at least one node device furthercomprises a learning engine operable to update a classification librarywith emerging protocols.
 10. The system of claim 1, wherein the at leastone node device further comprises a global positioning system (GPS). 11.The system of claim 1, wherein the at least one node device is operableto transmit an alert related to the at least one UV to a counter UVsystem.
 12. The system of claim 11, wherein the counter UV system isoperable to intercept communications between the at least one UV and acorresponding at least one UV controller.
 13. The system of claim 11,wherein the counter UAV system is configured with cameras for motiondetection.
 14. The system of claim 1, wherein the at least one nodedevice is operable to train the AI algorithm for UV recognition bycapturing and recording the RF data from a multiplicity of UVs overdifferent channels and different RF bandwidths.
 15. A system forunmanned vehicle (UV) recognition in a radio frequency (RF) environment,comprising: a multiplicity of node devices in network communication witha multiplicity of RF receivers; wherein the multiplicity of RF receiversis operable to capture the RF data in the RF environment, convert the RFdata to fast Fourier transform (FFT) data, and transmit the FFT data tothe multiplicity of node devices; wherein the multiplicity of nodedevices each comprises a signal processing engine, a detection engine, aclassification engine, and at least one artificial intelligence (AI)engine; wherein the signal processing engine is operable to average theFFT data into at least one tile; wherein the detection engine isoperable to identify at least one signal related to at least one UV;wherein the at least one AI engine is operable to generate an output foreach of the at least one tile to identify at least one UV; and whereinthe classification engine is operable to classify the at least one UV bycomparing the at least one signal to classification data.
 16. The systemof claim 15, wherein each of the multiplicity of node devices furthercomprises a state engine operable to control a flow of the at least onetile into the at least one AI engine.
 17. The system of claim 15,wherein the multiplicity of node devices is operable to estimate ageographical location for the at least one UV and/or determine avelocity of the at least one UV.
 18. A method for unmanned vehicle (UV)recognition in a radio frequency (RF) environment, comprising: providinga system comprising at least one node device in network communicationwith a multiplicity of RF receivers and a displaying device, whereineach of the at least one node device comprises a signal processingengine, a detection engine, and at least one artificial intelligence(AI) engine; the multiplicity of RF receivers capturing RF data in theRF environment, converting the RF data to fast Fourier transform (FFT)data, and transmitting the FFT data to the at least one node device; thesignal processing engine averaging the FFT data into at least one tile;and the detection engine identifying at least one signal related to atleast one UV in the at least one tile based on an AI algorithm.
 19. Themethod of claim 18, further comprising the at least one node devicetraining the at least one AI engine for UV recognition by capturing andrecording signals from a multiplicity of UVs over different channels anddifferent RF bandwidths.
 20. The method of claim 18, further comprisingtuning the multiplicity of receivers to a different frequency span.